Hand control interfaces and methods in virtual reality environments

ABSTRACT

A method for a user in a virtual reality environment to select an icon by virtually pressing the icon with the user&#39;s finger and displaying the icon as being pushed like a button in a real world environment when launching or activating a software application, setting or menu item represented by the icon.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 18/066,796, filed Dec. 15, 2022, titled, “HAND CONTROLINTERFACES AND METHODS IN VIRTUAL REALITY ENVIRONMENTS”, which is acontinuation of U.S. Non-Provisional application Ser. No. 17/830,109,now U.S. Pat. No. 11,531,448, filed Jun. 1, 2022, titled, “HAND CONTROLINTERFACES AND METHODS IN VIRTUAL REALITY ENVIRONMENTS”, the disclosuresof which are incorporated herein in their entirety.

BACKGROUND OF THE INVENTION

Virtual reality (VR) environments have grown more realistic andimmersive as VR headsets, controllers, interfaces, processor speeds,data storage and data transfer technologies have continued to improve.In electronic VR environments, such as implemented through developmenttools for the Oculus/Meta Quest platform (Oculus Platform SDK) by OculusVR (Irvine, CA) (parent company Meta) there are both challenges andopportunities for customizing how users interact with the VR environmentdepending on whether the user is operating a physical controller deviceor using their body, typically their hands.

In conventional VR systems, such as the Meta Quest 2 platform, users canuse either a physical controller or their hands to select and activateapplications that are typically displayed 2-dimensionally in anapplication selection interface. In such conventional VR systems, whenthe user is using a physical controller, they move the controller or usebuttons, joystick, scrolling or similar input controls on the controllerto select and launch an application from an application selectioninterface in which applications are displayed similar to applicationicons on mobile operating systems on tablets and phones, e.g., Androidand Apple iOS. If a user is using their hands to interact with theapplication interface in the conventional VR environment, whether byinitially using their hands or setting down the physical controller toswitch to hands, the VR application selection interface remains thesame; however, the user can make a pinch or similar pointing gesture tosimulate laser pointing from the fingertips of the hand recognized bythe system for selecting an application for launch from the display ofthe plurality of application icons in the VR application selectioninterface.

One of the deficiencies of the VR application selection interfaceremaining in the same display format between a user's use of a physicalcontroller and hands is that the current “laser pointing” handsexperience is that the laser pointing effect can be slow to berecognized or not even recognized, difficult to quickly and accuratelychoose an application for launch, and be less intuitive for movingbetween application selection and applications selection screens ascompared to use of the physical controller for which the applicationselection interface is more compatible.

SUMMARY OF THE INVENTION

Embodiments of the invention address these challenges by providingmethods and systems with improved display and functionality of anapplication selection interface in a VR environment that changesspecifically for use with a user's hands (or other body parts, e.g.,arms, feet, etc.) as compared to a separate and different applicationselection interface that is displayed when physical controllers areused. In embodiments, methods and systems of the invention areimplemented through development tools for the Oculus/Meta Quest platform(Oculus Platform SDK) by Oculus VR (Irvine, CA) (parent company Meta).It will be appreciated that the systems and methods, including relateddisplays, user interfaces, controls and functionalities, disclosedherein may be similarly implemented on other VR platforms with other VRSDKs and software development tools known to VR developers.

In an embodiment of the invention, a first application selectioninterface is provided in connection with use of a physical controller orcontrollers in a VR environment. The first selection interface may bethe same or similar to conventional VR application selection interfaces,such as implemented in the Meta Quest 2 VR system. When a user initiallydoes not use or switches away from using a physical controller, to use auser's hands in the VR environment, a second application selectioninterface that is better and more intuitively adapted for handinteraction and control is displayed to the user.

In further embodiments, a second and different application selectioninterface for use with a user's hands in a VR environment (as comparedto a first application selection interface for use with a physicalcontroller) does not require laser pointing functionality ofconventional VR platforms, but may be presented to appear to be directlyacted upon by the user hands, such as pushing or pulling the selectedapplication with a hand to launch the application.

In embodiments, application icons in the application selection interfaceused with hands control may be presented as three-dimensional buttonsthat a user may use their hands to press the “application button” tolaunch a desired application. In other embodiments application icons maybe presented a three-dimensional levers or switches that can be pulledor flipped to launch the desired application.

In further embodiments, scrolling arrows may also be presented asthree-dimensional buttons to a user for navigating the applicationselection interface by scrolling up and down by selecting thecorresponding up or down button, such as by pushing or pulling thescrolling arrow button. It will be appreciated that the applicationicons in the application selection interface will scroll up or downbased on selection of an up or down scrolling arrow.

In further embodiments, an application selection interface specific forhands control in a VR environment is displayed to appear at a differentdistance from the user in comparison to the application selectioninterface displayed for use with a physical controller. For example, aVR platform may calculate, either directly or indirectly based on armlength, relative arm length, user height, relative user height, or fromaverage user arm lengths or height, a preferred distance that VRapplication selection interface appears closer to the user than anapplication selection interface displayed when the user is using aphysical controller. In one embodiment, such distance for ahands-controlled application selection interface may be about half of anarm length (or relative or average determined arm length) for a user. Itwill be appreciated that in other embodiments, other distances from theuser to the hands-controlled application display interface may beimplemented for display, including user-selected distances.

In embodiments of the invention, an improved hands-control specificinterface may be implemented through (i) a software application runningon the VR platform operating system, (ii) programming of thehands-control interface as a feature in the VR platform operating systemand (iii) other programmed plug-ins and similar control modificationsadapted for use in the VR platform operating system. It will beappreciated that Oculus/Meta Quest system software based on Androidsource code provides one example of a VR platform operating system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of virtual reality application selectioninterface for control by a user's hands that interact with applicationicons and scrolling arrows to press and select an application for launchin an embodiment of the invention.

DETAILED DESCRIPTION

The present invention is directed to methods and interfaces forapplication selection in virtual reality environments, such as the MetaQuest 2 platform (Irvine, CA).

Typically, a user in a virtual reality platform wears a headset pairedwith controllers, such as “touch” controllers, wherein the userinteracts with their environment via the tracking and inputs receivedfrom one or both controllers to the computerized headset running systemor operating software. It will be appreciated that such interaction viacontrollers includes selection of application icons from an applicationselection interface when the user desires to launch and activate adesired application appearing in the selection interface.

A conventional alternative way for users to interact with a virtualreality environment in the Meta Quest 2 platform, includes handtracking. Hand tracking may be activated when a user puts down physicaldevice controllers or is not using the device controllers when beginninginteraction in the VR environment. Hand tracking works by using theinside-out cameras of the VR headset which detects the position andorientation of the user's hands and the configuration of the user'sfingers. Once detected, computer vision algorithms track the movementand orientation of the user's hands. Conventionally, the computer visionthen recognizes pinching and scrolling gestures from a user's hands thatappear and operate like “laser hands” for the user to select, scroll andprovide control inputs and interact with the virtual reality environmentaccording to the operating system software for the VR platform.

The hand tracking feature of the current Meta Quest 2 platform appliedto application selection, such as a typical application selectioninterface resembling application icons on mobile devices and tablets(e.g. IOS and android application selection screens) but in a 3D virtualreality environment can be difficult for users to achieve properpointing and recognition of the desired control input from the “laserhand” effect as compared to traditional procedures of tapping a buttonon an application selection interface of the mobile device and tabletplatforms.

Referring to FIG. 1 , an embodiment of the invention is illustratedwherein an alternative hand tracking and control interface for selectionof an application icon 25 to launch and activate an application isprovided in a VR environment to mimic the look and feel and of tappingor pressing an application icon from an application selection interfaceof mobile device and tablet environments.

When a user puts down a VR physical device controller or is not usingthe VR physical device controller when beginning a VR platformexperience, an application selection interface 10 includes a grid oficons of applications 20 displayed for a user's selection by a user'shand 100 that is tracked by the camera and computer vision of a VRheadset worn by the user. It will be appreciated that in embodiments ofthe invention, a hands-control specific interface of the invention maybe implemented through (i) a software application running on the VRplatform operating system, (ii) programming of the hands-controlinterface as a feature in the VR platform operating system and (iii)other programmed plug-ins and similar control modifications adapted foruse in the VR platform operating system. It will be appreciated thatOculus/Meta Quest system software based on Android source code providesone example of a VR platform operating system that may be modified toadd or provide an alternative of the hands-control interface of theinvention.

In a preferred embodiment, scroll buttons or representations aredisplayed as an up arrow 30 and a down arrow 35. The user's hands 100can use a finger to push those buttons 30 and 35 to scroll among theapplication icons 20 of the application selection interface 10 to choosean application for launching. To the user the selection of desiredapplication, such as education application icon 25, mimics pushing abutton in the real world, including pushing a scroll button 30 and 35 oran application icon 25.

In some embodiments, if the user returns to use of a physicalcontroller, the scroll buttons up 30 and down 35 may be programmed todisappear as push buttons since the button pushing experience would nolonger be relevant.

Unlike the pinching of fingers to provide a laser-pointing experience inprior art hands tracking of the Meta Quest 2 platform, a user of thepresent invention for hands control with an application selectioninterface simply moves their hand 100 and finger over a desiredapplication icon 25 among a plurality of application icons 20 displayedin the application selection interface 10, and simply pushes the desiredapplication icon 25 as a button to launch the application. The cameratracking and computer vision algorithm in the VR platform activates the“pressed” application when detected. To the user, the experience is moresimilar to pressing application icons in a mobile device or tableenvironment.

In embodiments of the invention, the application icons 20 could berepresented as 2-dimensional (2D) application icons that become 3Dbuttons. When a user is in the hands-control interface of the presentinvention, the application icons 20 can lift up from 2D display to beingdisplayed as a 3D button, so that it is clear to a user that they canpush the application icon 20 as a button down with the user's finger onthe user's hand 100.

In some embodiments, a user does not need to set down controllers orgesture to activate the hands-control and application selectioninterface of the invention, but could choose from a settings display,such as at the bottom of the user's VR environment display and selectactivation of the desired hands-control feature.

In further embodiments, a user could select menus and settings featuresthat are not necessarily applications in a similar manner ofhands-control selecting an icon representing a menu item or setting. Forexample, a Wi-Fi settings button might change from a 2D to 3D display,so a user knows that the setting can be activated as pressed from theuser's hands 100 instead of physical device controllers.

In some embodiments, pulling and sliding actions from a user's hand 100might be implemented in the VR platform instead of button pressingdepending on the desired action of the user. For example, a user couldgrab an application icon 20 or between application icons and move theuser's hand up or down to cause the application grid to scroll up ordown respectively. That is, the user user's their hand 100 to grabsomething that is displayed to the user and move it up or down, like howsomeone might grab a huge classroom chalkboard or window to slide it upor down in the real world.

In further embodiments of the invention, when a user is in thehands-control interface for application selection, the interfaceapplications grid and settings button appears to move toward the userafter calculating about how long the user's arms are—such as (i) basedon average American height or; (ii) based on how far away physicalcontrollers were when dangling at the user's side or; (iii) based on auser's height since that dimension can be used determine arm length. Itwill be appreciated that other methods for calculating a preferreddistance for an application selection interface 10 to be displayedrelative to the user hands 100 may be utilized, with the intention thatthe user perceives an ability to reach and press or grab or otherwiseinteract with the application icons 20 or similar menus and settingsfeature and buttons as an alternative to controller operation andlaser-hands interaction.

Various embodiments of the invention have been described. It will,however, be evident that various modifications and changes may be madethereto, and additional embodiments may be implemented, withoutdeparting from the broader scope of the invention as set forth by theclaims. This specification is to be regarded in an illustrative ratherthan a restrictive sense.

What is claimed is:
 1. A method for providing an interactive controlinterface in a virtual reality environment comprising: detecting that auser using a virtual reality environment via a virtual reality hardwaredevice is using their hands to interact with the interactive controlinterface including an icon that appears as a button that represents atleast one of a software application, setting and a menu item; receivinga pressing action from a tracked finger of the user to the icon;displaying the icon in the virtual reality environment to mimic theappearance of a button being pushed in a real world environment; andlaunching or activating the corresponding software application, settingor menu item represented by the icon after receiving the pressing actionfrom the tracked finger.
 2. The method of claim 1, further comprisingdisplaying the interactive control interface to appear within arm lengthdistance of the user.
 3. The method of claim 1, further comprisingdisplaying the interactive control interface to appear at auser-selected distance.
 4. The method of claim 1, further comprisingcalculating an interface display distance based on at least one of armlength and height of the user and displaying the interactive controlinterface to appear at the interface display distance.
 5. The method ofclaim 1, further comprising displaying the interactive control interfaceto appear at a distance based on at least one of an average arm lengthand height of a plurality of users of the virtual reality environment.6. The method of claim 1, further comprising detecting when a userdiscontinues use of a controller to interact with the virtual realityenvironment and changing appearance of the interactive control interfaceto appear closer to the user when using their hands to interact with theinteractive control interface.
 7. The method of claim 6, furthercomprising displaying the interactive control interface to appear closerwithin arm length distance of the user.
 8. The method of claim 6,further comprising displaying the interactive control interface toappear closer at a user-selected distance.
 9. The method of claim 6,further comprising calculating a closer interface display distance basedon at least one of arm length and height of the user and displaying theinteractive control interface to appear at the closer interface displaydistance.
 10. The method of claim 1, further comprising displaying theinteractive control interface to appear at a closer distance based on atleast one of an average arm length and height of a plurality of users ofthe virtual reality environment.
 11. The method of claim 1, furthercomprising detecting when a user discontinues use of a controller tointeract with the virtual reality environment and changing appearance ofthe icon from a 2-dimensional representation of a button to a3-dimensional representation of a button.
 12. The method of claim 1,further comprising detecting a pulling action from a hand of the userand displaying a scrolling of the icon to correspond to a direction ofthe pulling action.
 13. The method of claim 1, further comprisingdetecting a sliding action from a hand of the user and displaying ascrolling of the icon to correspond to a direction of the slidingaction.
 14. The method of claim 1, further comprising displaying one ormore scroll buttons adjacent the icon and receiving a selection of ascroll button and displaying a scrolling of the icon to correspond to adirection of the scroll button that is selected.
 15. The method of claim14, wherein the one or more scroll buttons include one or both of an uparrow button and a down arrow button.
 16. The method of claim 14,wherein the icon appears in a grid of a plurality of icons of theinteractive control interface.
 17. The method of claim 1, wherein theicon appears in a grid of a plurality of icons of the interactivecontrol interface.
 18. The method of claim 6, wherein the icon appearsin a grid of a plurality of icons of the interactive control interface.19. The method of claim 2, wherein the icon appears in a grid of aplurality of icons of the interactive control interface.
 20. The methodof claim 9, wherein the icon appears in a grid of a plurality of iconsof the interactive control interface.